feat(recalibration): a temperature per predicted label, when one temperature is the wrong shape - #103

Fixes #4. This follows the approved design.

What it does

When it runs. Only after the global fit ends in one of the two wrong-shape verdicts: no_material_improvement (refused) or residual_above_floor (partial). The global path is unchanged and always runs first. When the model is already calibrated or the interval spans 1.0, the fallback never runs.

The method. recalibrate_per_label() fits one temperature per predicted label:

• It takes the vendor's pick as predicted_labels, not an argmax, since the two differ on a tie.
• It reuses the global fit's split, and a test checks the held-out rows are identical.
• Each label is fitted on its own fit rows, in the global fit's form (multiclass or binary), with its own bootstrap interval.
• Each held-out row is scaled by its own label's temperature. Scaling never changes the top label, so rows stay in their group.

The verdict. The global result's verdict logic moves to a shared _Gate, so both fits reach recommended, partial or refused by exactly the same rule.

Row gate. A label with fewer than 100 fit rows (DEFAULT_MIN_LABEL_ROWS) is left as it came, and the report names it with its counts.

The report

A #### Per-label fallback block under Recalibration, in a real render:

• One temperature did not fit, so one temperature per predicted label was fitted on the same split: 2000 fit rows and 2000 held-out rows (seed 0), multiclass form. These are a different correction from the global temperature, one parameter per label, and are not comparable with it.

label	fit rows	held-out rows	T	95 percent interval
billing	493	492	2.088	[1.853, 2.338]
other	506	514	1.032	[0.905, 1.148]

• ECE 0.0639 before, 0.0191 after, against a calibrated-model floor of 0.0180 (95th percentile 0.0278) on the held-out rows.
• Recommended: post-scaling ECE is inside the floor.
• Apply: the per-label temperatures, each to the rows the model predicts that label for, leaving any label not fitted as it came. Not the global temperature.
• The cascade below scores on the global temperature when one was emitted, otherwise on the probabilities as they came; it does not use these.

The global fit on that same run was partial, at 1.6 times the floor.

A refusal prints row counts only, no temperatures and no ECE, following the existing rule that a refused fit shows no number anyone could lift. The Apply line picks per-label if it isn't refused and leaves less behind, or when the global fit was refused. Otherwise it picks the partial global temperature, and otherwise neither.

Evidence

scripts/per_label_study.py runs the mock at 5 seeds. Each figure is held-out ECE over the floor's 95th percentile:

• Per-label bias: from about 100 fit rows per label, per-label lands inside the floor, while one temperature falls further behind as rows grow.
• Global skew: the extra parameters cost 5 to 15% more held-out ECE. The report never pays that cost, because it only tries per-label after a wrong-shape verdict.
• The gate: a gate of 40 also worked at 500 rows on the mock (0.69x, shipped on 2 of 5 seeds). The default stays at 100 because the mock is per-label scaling's best case. METHODOLOGY's new section says so.

Also

• The global refusal text no longer says "plumbline fits neither".
• README's limitation line and PLAN's v0.2 list are updated. PLAN's stale "adapters do not receive label_descriptions" item is now marked landed in v0.1.1 (label_descriptions are parsed and then dropped #39).
• Out of scope, as the issue asks: vector and matrix scaling, gold-label keying, and per-label temperatures in the cascade.

Tests: tests/test_per_label_recalibration.py, 7 tests covering recovery against the global fit, which labels get corrected, the identical split, the row gate, the honest-model refusal, and the report block appearing (and not appearing). The full suite is 606 passed. docs/example-report.md is unchanged. The gate, the site checks and the prose checks all pass.

🤖 Generated with Claude Code